3d video processing

ABSTRACT

A method, an apparatus, and a non-transitory computer readable medium for performing 2D to 3D conversion are presented. A 2D input source is extracted into left and right 3D images. Motion vectors are calculated for the left and right 3D images. Frame rate conversion is performed on the left 3D image and the right 3D image, using the respective calculated motion vectors, to produce motion compensated left and right 3D images. The left and right 3D images and the motion compensated left and right 3D images are reordered for display.

CROSS REFERENCE TO RELATED APPLICATION

This application is a divisional of U.S. patent application Ser. No.12/397,448, filed Mar. 4, 2009, the entire contents of which is herebyincorporated by reference as if fully set forth herein.

FIELD OF INVENTION

The present invention relates generally to video processing, and moreparticularly, to a method and apparatus for processing 3D video.

BACKGROUND

Current 3D content is based on a film source, which has a rate of 24frame per second. Television broadcasts are at either a 50 Hz or a 60 Hzrefresh rate. To convert a film source for television broadcast, thereneeds to be a frame rate conversion. This is typically accomplished by aframe rate conversion (FRC) process.

One way of performing FRC on a film source to a 60 Hz refresh rate is touse a 2:3 pulldown, in which a first frame is displayed twice and asecond frame is displayed three times. This is repeated for each pair offrames in the film source. Because the frame rate for television is notthe same as that for film, there is a motion judder problem (i.e., theviewer may notice that motion within the image is not smooth) introducedduring the 2:3 pulldown.

OCCLUSION PROBLEM

The output of the FRC process is a new (middle) frame that is placedbetween successive source frames. A problem with the new frame can ariseif there is missing information regarding an obscured or hidden objectin the background. If there is missing information, FRC cannotaccurately produce the middle frame. This is referred to as theocclusion problem. If the middle frame is produced by motion estimation(determining a motion vector) and motion compensation (applying themotion vector to reduce the effects of motion), i.e., by the FRCprocess, then some immaterial images may be produced. This problemexists in both 2D images and in 3D images, where the hidden material canbe in the left image, the right image, or both.

An example of an occlusion region is shown in FIG. 1, which shows fiveframes—a current frame (N), two previous frames (N−2, N−1), and twolater frames (N+1, N+2). In normal FRC, an occlusion region shows nogood matching motion vectors. If there is a motion vector fielddiscontinuity in a homogeneous region, this could lead to potentialocclusion regions because the correct motion vector could not beestimated due to the discontinuity. The motion vector discontinuity dueto occlusion could be detected be comparing the motion vectors betweenframes N−2 and N−1 and between frames N−1 and N+1. To be able to handlethe occlusion region correctly, more than four consecutive frames ofdata will be needed, creating a complex solution.

There is therefore a need to reduce judder when converting a filmsource, in particular with a 3D film source. There is also a need toaddress the occlusion problem in 3D sources.

SUMMARY

The judder problem can be addressed by separating a 3D image into itscomponent left image and right image, performing FRC on the individualimages, and then reordering the images for display. The separation andFRC processes can be reversed, in that FRC can be performed on the 3Dimage and the original and motion compensated images can be separatedinto their component left and right images.

One way to address the occlusion problem in 3D sources is to utilize thedisparity information (the difference between the left image and theright image) to help generate the hidden information.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed understanding of the invention may be had from thefollowing description, given by way of example, and to be understood inconjunction with the accompanying drawings, wherein:

FIG. 1 is a diagram showing an example of an occlusion region;

FIG. 2 is a flowchart of a method for performing FRC on a 3D inputsource;

FIG. 3 is a diagram showing an example application of the method of FIG.2;

FIG. 4 is a flowchart of an alternate method for performing FRC on a 3Dinput source;

FIG. 5 is a diagram showing an example application of the method of FIG.4;

FIG. 6 is a flowchart of a method for performing 2D to 3D conversion;

FIG. 7 is a diagram showing an example application of the method of FIG.6;

FIG. 8 is a flowchart of an alternate method for performing 2D to 3Dconversion;

FIG. 9 is a diagram showing an example application of the method of FIG.8;

FIG. 10 is a block diagram of an apparatus configured to perform any ofthe methods of FIG. 2, 4, 6, or 8;

FIG. 11 is a diagram showing how 3D images assist in addressing theocclusion problem;

FIG. 12 is a diagram showing an example of disparity between what aviewer's left eye and right eye sees in a 3D image;

FIG. 13 is a diagram showing how background information in a 3D imagecan be obtained by using the disparity; and

FIG. 14 is a flowchart of a method for addressing the occlusion problem.

DETAILED DESCRIPTION

One way to reduce the judder problem is via proper application of FRC.This can be achieved by using motion estimation and motion compensationduring the FRC process. With a 3D input source, the motion estimationand motion compensation needs to be performed on both the left image andthe right image of a 3D video frame. The left image and the right imageneed to be separated at some point during the process; the separationcan occur before FRC (as shown in FIGS. 2 and 3) or after FRC (as shownin FIGS. 4 and 5).

It is noted that the computational complexity is approximately the samefor both methods. With a 3D image, each left image and right image ishalf the size of a full frame. Processing (e.g., performing FRC) on asmaller size image is easier, and the method shown in FIG. 2 maycomplete faster than the method shown in FIG. 4.

Separation, then FRC

FIG. 2 is a flowchart of a method 200 for performing FRC on a 3D inputsource. A 3D frame is separated into a left image and a right image(step 202). A motion vector is calculated for the left image (step 204)and for the right image (step 206). It is noted that calculating themotion vectors for the left image and the right image (steps 204 and206) can be reversed.

It is further noted that only one motion vector may be calculated, foreither the left image or the right image, and that motion vector can beapplied to the other image. The reasoning behind this possibility isthat in most cases (approximately 95% on the time), the motion vectorbetween the left image and the right image is very similar. A slightdifference between the respective motion vectors (if calculatedseparately) should not have an effect. If the motion vectors between theleft image and the right image are substantially different, then thedisparity will be different and the motion vectors should be calculatedseparately. By calculating only one motion vector, the hardwarecomplexity can be reduced; the determination whether only one motionvector is calculated can be based on the hardware present and isimplementation-specific.

One exceptional case to note is when an object in the image isapproaching only one eye. At a distance close to the viewer, there willbe a large disparity between the images, and there will also be a largedifference in the motion vectors. This is because the left image and theright image have different motions associated with them, since theobject is moving in a different direction and at a different distancefrom each eye.

FRC is then performed on the left image (step 208) and on the rightimage (step 210). Performing the FRC (steps 208 and 210) can also bereversed, provided that the motion vectors are determined before FRC isperformed. After FRC has been performed on both the left image and theright image, the converted images are reordered for display (step 212).

FIG. 3 is a diagram showing an example application of the method of FIG.2. A 3D frame 300 is separated into a left image 302 and a right image304. A motion vector (MV₁) 310 is calculated for the left image 302 andis applied to determine a motion compensated image (L2) 320. A motionvector (MV₂) 312 is calculated for the right image 304 and is applied todetermine a motion compensated image (R2) 322. The original images 302,304 and the motion compensated images 320, 322 are then reordered fordisplay.

FRC, Then Separation

FIG. 4 is a flowchart of an alternate method 400 for performing FRC on a3D input source. A motion vector is calculated for the 3D image (step402) and FRC is performed on the 3D image (step 404). Both the original3D image and the motion compensated image are separated into left andright images (step 406). All of the images are then reordered fordisplay (step 408).

FIG. 5 is a diagram showing an example application of the method 400. A3D frame 500 includes a left image 502 and a right image 504. FRC isperformed on the 3D frame 500, to generate a motion compensated frame510, including a motion compensated left image (L2) 512 and a motioncompensated right image (R2) 514. The original 3D image 500 and themotion compensated 3D image 510 are then separated into their respectiveleft and right images 502, 504, 512, 514 and the individual images arereordered for display.

2D to 3D extraction, then FRC

FIG. 6 is a flowchart of a method 600 for performing 2D to 3Dconversion. A 2D image is extracted into a 3D image, including a leftimage and a right image (step 602). A motion vector is calculated forthe left image (step 604) and for the right image (step 606). It isnoted that calculating the motion vectors for the left image and theright image (steps 604 and 606) can be reversed. It is further notedthat only one motion vector may be calculated, for either the left imageor the right image, and that motion vector can be applied to the otherimage.

FRC is then performed on the left image (step 608) and on the rightimage (step 610). Performing the FRC (steps 608 and 610) can also bereversed, provided that the motion vectors are determined before FRC isperformed. After FRC has been performed on both the left image and theright image, the converted images are reordered for display (step 612).

FIG. 7 is a diagram showing an example application of the method of FIG.6. Two 2D images 702, 704 are extracted into 3D images with left andright images 710, 712, 720, 722. FRC is then performed on the left andright images to produce motion compensated images L1* 714, R1* 716, L2*724, and R2* 726. The original images 710, 712, 720, 722 and the motioncompensated images 714, 716, 724, 726 are reordered for display.

FRC, Then 2D to 3D Extraction

FIG. 8 is a flowchart of an alternate method 800 for performing 2D to 3Dconversion. A motion vector is calculated for a 2D image (step 802) andFRC is performed on the 2D image (step 804). The original 2D image andthe motion compensated 2D image are extracted into respective 3D images,including left images and right images (step 806).

FIG. 9 is a diagram showing an example application of the method of FIG.8. FRC is performed on two 2D images 902, 904 to produce motioncompensated images 1* 910 and 2* 912. The original 2D images 902, 904and the motion compensated 2D images 910, 912 undergo 3D extraction intorespective left and right images, to generate left and right images920-934.

Apparatus Configured to Perform the Methods

FIG. 10 is a block diagram of an apparatus 1000 configured to performany of the methods 200, 400, 600, or 800. While the apparatus 1000 asshown is capable of performing any of the methods 200, 400, 600, or 800,the apparatus 1000 can be specifically configured to perform only one ofthe methods be removing unused components.

The apparatus 1000 includes a 2D to 3D image extracting device 1002, amotion vector calculating device 1004, a 3D image separating device1006, a frame rate conversion device 1008, and an image reorderingdevice 1010.

When performing the method 200, the apparatus 1000 operates as follows.A 3D input 1020 is provided to the 3D image separating device 1006,which separates the 3D input 1020 into left and right images. Motionvectors for the left and right images are calculated by the motionvector calculating device 1004. The frame rate conversion device 1008performs FRC on the left and right images using the calculated motionvectors to produce motion compensated images. The left and right imagesand the motion compensated images are passed to the image reorderingdevice 1010, where the images are reordered for display and are output1022.

When performing the method 400, the apparatus 1000 operates as follows.A 3D input 1030 is provided to the motion vector calculating device1004, which calculates the motion vector for the 3D input 1030. Theframe rate conversion device 1008 performs FRC on the 3D input 1030using the calculated motion vector to produce a motion compensated 3Dimage. The 3D input 1030 and the motion compensated 3D image are passedto the 3D input 1030 separating device 1006, which generates left andright images from the 3D image and left and right images from the motioncompensated 3D image. The original left and right images and the motioncompensated left and right images are passed to the image reorderingdevice 1010, where the images are reordered for display and are output1022.

When performing the method 600, the apparatus 1000 operates as follows.A 2D input 1040 is provided to the 2D to 3D image extracting device1002, which extracts the 2D input 1040 into left and right 3D images.The motion vector calculating device 1004 calculates motion vectors forthe left and right images. The frame rate conversion device 1008performs FRC on the left and right images using the calculated motionvectors to produce motion compensated left and right images. The leftand right images and the motion compensated left and right images arepassed to the image reordering device 1010, where the images arereordered for display and are output 1022.

When performing the method 800, the apparatus 1000 operates as follows.A 2D input 1050 is provided to the motion vector calculating device1004, which calculates the motion vector for the 2D input 1050. Theframe rate conversion device 1008 performs FRC on the 2D input 1050using the calculated motion vector to produce a motion compensatedimage. The 2D input 1050 and the motion compensated image are passed tothe 2D to 3D image extracting device 1002, which extracts the 2D input1050 and the motion compensated image into left and right 3D images andproduces an output 1052.

Addressing the Occlusion Problem

The occlusion problem in 3D images can be addressed by using both theleft image and the right image together during FRC. The two images maycontain the information missing from a 2D image due to the differentinformation contained in the left image and the right image (alsoreferred to as disparity information).

FIG. 11 is a diagram showing how 3D images assist in addressing theocclusion problem. The viewer's left eye is at a first coordinate d₁along the X-axis and the viewer's right eye is at a second coordinate d₂along the X-axis. There is a viewing angle θ between the left and righteyes relative to the plane z₁. By changing the angle θ or the positionof the left or right eye (d₁ or d₂), more information can be obtainedfor images in the background (plane z₂) as compared to images in theforeground (plane z₁). The additional information in a 3D image (onplane z₂) can be used to address the occlusion problem.

Generally, motion estimation in block matching can be stated as, forexample, the calculation of the minimization of the mean absolutedifference (MAD) between the viewer's left and right eyes (d₁ and d₂).

$\begin{matrix}{{{MAD}( {d_{1},d_{2}} )} = {\frac{1}{N_{1}N_{2}}{\sum\limits_{{({{n\; 1},{n\; 2}})} \in B}\; \lbrack {{s( {n_{1},n_{2},k} )} - {s( {{n_{1} + d_{1}},{n_{2} + d_{2}},{k + 1}} )}} \rbrack}}} & {{Equation}\mspace{14mu} (1)} \\{\lbrack {d_{1},d_{2}} \rbrack^{T} = {\arg \; {\min\limits_{({d_{1},d_{2}})}{{MAD}( {d_{1},d_{2}} )}}}} & {{Equation}\mspace{14mu} (2)}\end{matrix}$

where B denotes an N₁×N₂ block for a set of candidate motion vectors(d₁, d₂) and signal s is at a pixel (n₁, n₂) in frame k (shown in FIG.11 as frame Z₁). T represents the transpose of the motion vectors d₁,d₂.

FIG. 12 is a diagram showing an example of disparity between what aviewer's left eye and right eye sees in a 3D image. At time t, theviewer can see at least a portion of block A 1202, block B 1204, andblock C 1206. Based on the relative positioning of block C 1206 betweenblock A 1202 and block B 1204, the viewer can see more of block C 1206with their left eye than with their right eye.

At time t+1, block B 1204 has moved to the right and the viewer can seemore of block C 1206 with their right eye. Ideally, when the motionvector from time t to time t+1 is comparable to the disparity, the angleθ can be defined as:

$\begin{matrix}{\theta \approx {2\; {\tan^{- 1}( \frac{d^{*}}{z} )}}} & {{Equation}\mspace{14mu} (3)} \\{d^{*} = {d^{*}( \overset{arrow}{\upsilon} )}} & {{Equation}\mspace{14mu} (4)} \\{{Disparity} \approx \frac{2d}{z}} & {{Equation}\mspace{14mu} (5)}\end{matrix}$

where v is a motion vector, d is a disparity vector, and d* is thedistance between the viewer's left eye view and the viewer's right eyeview.

FIG. 13 is a diagram showing how background information in a 3D imagecan be obtained by using the disparity. The covered portion in the image(block C 1206) is handled in a similar manner as an occlusion is handledin a normal FRC process. In frame N−1, there is an occlusion region1302. Continuing the example from FIG. 12, the occlusion region 1302 isthe amount of block C 1206 that is hidden at time t. In the currentframe N, there is an occlusion region 1304 (the amount of block C 1206that is hidden at time t+1). Motion vectors are calculated by using thenormal FRC process, based on frames N−1 and N+1. The motion vectors arecorrected based on the disparity information (via a disparity to motionvector conversion). Motion compensation is then performed based on thecorrected motion vectors.

FIG. 14 is a flowchart of a method 1400 for addressing the occlusionproblem. A stereo stream (the left and right images of a 3D videostream) is provided (step 1402). A disparity analysis is performed onthe received stream (step 1404). The motion vector is estimated based onthe result of the disparity analysis (step 1406) and motion compensationis performed, utilizing the motion vector (step 1408).

FRC is then performed on the 2D stream (step 1410). This step includesseparately performing FRC on the left image and the right image (each a2D stream). 2D FRC can also be used in cases where there is a 3D input,but only a 2D display, and therefore, the FRC output needs to be a 2Dstream.

The present invention can be implemented in a computer program tangiblyembodied in a non-transitory computer-readable storage medium containinga set of instructions for execution by a processor or a general purposecomputer; and method steps can be performed by a processor executing aprogram of instructions by operating on input data and generating outputdata. Suitable processors include, by way of example, both general andspecial purpose processors. Typically, a processor will receiveinstructions and data from a read-only memory (ROM), a random accessmemory (RAM), and/or a storage device. Storage devices suitable forembodying computer program instructions and data include all forms ofnon-volatile memory, including by way of example semiconductor memorydevices, magnetic media such as internal hard disks and removable disks,magneto-optical media, and optical media such as CD-ROM disks anddigital versatile disks (DVDs). In addition, while the illustrativeembodiments may be implemented in computer software, the functionswithin the illustrative embodiments may alternatively be embodied inpart or in whole using hardware components such as Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), orother hardware, or in some combination of hardware components andsoftware components.

While specific embodiments of the present invention have been shown anddescribed, many modifications and variations could be made by oneskilled in the art without departing from the scope of the invention.The above description serves to illustrate and not limit the particularinvention in any way.

What is claimed is:
 1. A method for performing 2D to 3D conversion,comprising: extracting a 2D input source into left and right 3D images;calculating motion vectors for the left and right 3D images; performingframe rate conversion on the left 3D image and the right 3D image, usingthe respective calculated motion vectors, to produce motion compensatedleft and right 3D images; and reordering the left and right 3D imagesand the motion compensated left and right 3D images for display.
 2. Anon-transitory computer-readable storage medium storing a set ofinstructions for execution by a general purpose computer to perform 2Dto 3D conversion, the set of instructions comprising: an extracting codesegment for extracting a 2D input source into left and right 3D images;a calculating code segment for calculating motion vectors for the leftand right 3D images; a frame rate conversion code segment for performingframe rate conversion on the left 3D image and the right 3D image, usingthe respective calculated motion vectors, to produce motion compensatedleft and right 3D images; and a reordering code segment for reorderingthe left and right 3D images and the motion compensated left and right3D images for display.
 3. The non-transitory computer-readable storagemedium according to claim 2, wherein the instructions are hardwaredescription language (HDL) instructions used for the manufacture of adevice.
 4. An apparatus configured to perform 2D to 3D conversion,comprising: a 2D to 3D image extracting device, configured to extract a2D input source into left and right 3D images; a motion vectorcalculating device, configured to calculate motion vectors for the leftand right 3D images; a frame rate conversion device, configured toproduce motion compensated left and right 3D images based on the inputleft and right 3D images and the calculated motion vectors; and an imagereordering device, configured to reorder the left and right 3D inputimages and the motion compensated left and right 3D images for display.5. A non-transitory computer-readable storage medium storing a set ofinstructions for execution by one or more processors to facilitatemanufacture of an apparatus configured to perform 2D to 3D conversion,the apparatus comprising: a 2D to 3D image extracting device, configuredto extract a 2D input source into left and right 3D images; a motionvector calculating device, configured to calculate motion vectors forthe left and right 3D images; a frame rate conversion device, configuredto produce motion compensated left and right 3D images based on theinput left and right 3D images and the calculated motion vectors; and animage reordering device, configured to reorder the left and right 3Dinput images and the motion compensated left and right 3D images fordisplay.
 6. A method for performing 2D to 3D conversion, comprising:calculating a motion vector for a 2D input source; performing frame rateconversion on the 2D input source, to produce a motion compensated 2Dimage; and extracting the 2D input source and the motion compensated 2Dimage into left and right 3D images.
 7. A non-transitorycomputer-readable storage medium storing a set of instructions forexecution by a general purpose computer to perform 2D to 3D conversion,the set of instructions comprising: a calculating code segment forcalculating a motion vector for a 2D input source; a frame rateconversion code segment for performing frame rate conversion on the 2Dinput source, to produce a motion compensated 2D image; and anextracting code segment for extracting the 2D input source and themotion compensated 2D image into left and right 3D images.
 8. Thenon-transitory computer-readable storage medium according to claim 7,wherein the instructions are hardware description language (HDL)instructions used for the manufacture of a device.
 9. An apparatusconfigured to perform 2D to 3D conversion, comprising: a motion vectorcalculating device, configured to calculate a motion vector; a framerate conversion device, configured to produce a motion compensated imagebased on an input image and the calculated motion vector; and a 2D to 3Dimage extracting device, configured to extract a 2D input source intoleft and right 3D images.
 10. The apparatus according to claim 9,wherein the motion vector calculating device is further configured tooperate on a 2D input source.
 11. The apparatus according to claim 10,wherein the 2D to 3D image extracting device is further configured toextract the 2D input source and the motion compensated 2D image intoleft and right 3D images.
 12. A non-transitory computer-readable storagemedium storing a set of instructions for execution by one or moreprocessors to facilitate manufacture of an apparatus configured toperform 2D to 3D conversion, the apparatus comprising: a motion vectorcalculating device, configured to calculate a motion vector; a framerate conversion device, configured to produce a motion compensated imagebased on an input image and the calculated motion vector; and a 2D to 3Dimage extracting device, configured to extract a 2D input source intoleft and right 3D images.
 13. The non-transitory computer-readablestorage medium according to claim 12, wherein the motion vectorcalculating device is further configured to operate on a 2D inputsource.
 14. The non-transitory computer-readable storage mediumaccording to claim 13, wherein the 2D to 3D image extracting device isfurther configured to extract the 2D input source and the motioncompensated 2D image into left and right 3D images.